Implement subresources in VWs - #4342
Conversation
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
60af7fe to
3fc8c0e
Compare
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
127 because a resource can have up to 63 bytes per segment, so 63 for the resource, 63 for the subresource and 1 for the `/`. Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
3fc8c0e to
4aba907
Compare
|
/retest |
Signed-off-by: Nelo-T. Wallus <n.wallus@sap.com>
|
/retest One is #4343 |
|
/retest infra failure |
|
/retest |
|
Nice! /lgtm I'll leave approval for a bit if someone else wants to have a look. |
|
LGTM label has been added. DetailsGit tree hash: 0bb5af6988e649277b698d6e26a194d0f83dfff4 |
| // subresource status is implicitly granted with its parent resource. | ||
| // others must be claimed explicitly |
There was a problem hiding this comment.
Yes, we just never checked for subresource permissions because only implemented status so far.
Given kube also doesn't give implicit access to the status subresource - but requiring this now would break existing APIExport/-Bindings.
I wanted to bring this up in todays community meeting to see how we feel about this.
On one hand I'd rather we are exact, on the other I'm not sure how we could make this "nice" for end users.
E.g. we could gate the implicit status on a feature gate, but that isn't really nice.
We could just keep status being implicit for now and require it explicitly in the next API version.
But I'm not really a fan of either.
There was a problem hiding this comment.
Albeit not on paper, but the "spirit" of offering a service is for the consumer to give the spec, and provider to update the status - so I think it's assumed the provider needs access to that at all times, even when claiming. So if we're looking for an excuse to keep these perms implicit, I think this could be one :D but it does sound a bit weak.
There was a problem hiding this comment.
I mean yeah but for resources exported via an APIExport we are already implicitly claiming the resource and the subresources^^
But I get what you mean. Plus that subresources are only really interesting for update/patch, so at that point the consumer is already accepting mutations from the provider on the claimed resource - and that could include subresources.
| @@ -119,6 +124,50 @@ func provideDelegatingRestStorage(ctx context.Context, dynamicClusterClientFunc | |||
|
|
|||
| // TODO(sttts): add scale subresource | |||
There was a problem hiding this comment.
Is this TODO still relevant?
There was a problem hiding this comment.
Yes. CRDs can support the scale subresource, so technically a CRD could be exported via an APIExport that supports the scale subresource.
https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#scale-subresource
There was a problem hiding this comment.
But why is it different than status? Isn't this PR working for all subresources?
There was a problem hiding this comment.
Not all, but making it possible to handle subresources other than status with the changes to permission claims and handling it in authorization.
I can take a stab at implementing the scale subresource as well for completeness :D
I'm not sure if scale is actually different or if sttts just didn't bother with it at the time.
But I think there's more work down the line because I'm not sure how well this works for e.g. virtual resource. But I haven't used virtual resources at all so far so 🤷
There was a problem hiding this comment.
Done :D Scale subresource works!
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
|
New changes are detected. LGTM label has been removed. |
mjudeikis
left a comment
There was a problem hiding this comment.
Few other comments:
Two silent no-ops slip through admission:
an export claiming only serviceaccounts/token without claiming serviceaccounts → no API served, no error, no condition;
a CRD claim for a nonexistent subresource (cowboys/notreal + identityHash) → accepted, dead.
I have more things, but I think all and all we need better e2e:
- Non existing resource
- Only status claim
- Inherited status permissions (status does not have claims)
....
| - verbs | ||
| type: object | ||
| x-kubernetes-validations: | ||
| - message: subresource claims must use selector.matchAll |
There was a problem hiding this comment.
Does this meant that if I claim named resource virtualmachine I will get all other virtualmachines subresources?
There was a problem hiding this comment.
It just means that you can apply a selector on top of the selectors for the parent resource.
e.g. you have the virtualmachines resource with label my.exposed.resources=true and then you claim the subresource virtualmachines/ssh you cannot apply another label filter on top.
I had that implemented and I see some use for it (I mentioned that in the description) but I don't think that too many people will use it.
The story is basically "when you claim a subresource you must have access to its parent".
Hence the CEL validation that the selector must be matchall.
| ab.SetLabels(lbls) | ||
| } | ||
|
|
||
| // subresource claims have no selector, set a matchAll so users |
There was a problem hiding this comment.
This sets MatchAll = trueunconditionally, even when the user explicitly set matchLabels/matchExpressions on the subresource claim. ValidateAPIBindingPermissionClaims then rejects with "matchLabels cannot be used with matchAll" — for a matchAll the user never wrote, which will be very confusing to debug :)
There was a problem hiding this comment.
The user cannot set matchlabels/-expression; I added a CEL rule for that.
Maybe the rule doesn't hold in which case I have to fix that :D
| // not provided by an api export. | ||
| // | ||
| // +kubebuilder:validation:Pattern=`^[a-z][-a-z0-9]*[a-z0-9]$` | ||
| // +kubebuilder:validation:Pattern=`^[a-z][-a-z0-9]*[a-z0-9](/[a-z][-a-z0-9]*[a-z0-9])?$` |
There was a problem hiding this comment.
Bit of AI explaining as it did better job than I did. I think this is important one , if not most. I tried to look to this via "controller-runtime generated patterns"
What the API now allows:
The new regex on resource accepts anything shaped like foo/bar — including cowboys/status. Nothing anywhere rejects .../status as a claim. So you can write it in an APIExport and APIBinding, and it gets accepted and stored.
What the authorizer actually does with status requests:
When a request comes in for a status subresource, the authorizer deliberately ignores the subresource and looks for a claim on the parent resource only:
// binding.go
claimedResource := normalizedGR.Resource // "cowboys"
if sub != "" && sub != "status" { // status is skipped!
claimedResource = resource + "/" + sub
}
So for a status update, it searches your claims for cowboys — never for cowboys/status.
The trap:
Imagine a provider who thinks in RBAC terms (which the docs literally encourage — "in the style of RBAC rules"). In RBAC you'd write:
cowboys → verbs get, list (read the objects)
cowboys/status → verbs update (write only the status)
That's a completely reasonable, even best-practice layout. But here's what happens:
- Both claims are accepted. No error, no warning. ✅ (looks fine)
- Provider tries to update a cowboy's status.
- Authorizer looks for a claim on cowboys with verb update.
- The cowboys claim only has get, list → denied.
- The cowboys/status claim with update? Never consulted. It's a dead claim — it exists, it's accepted, and it does absolutely nothing.
- The user gets a permission denied and has zero indication that their cowboys/status claim is being ignored. They'll stare at it and think "but I claimed it right there!"
Why "accepting-but-ignoring is the worst of both":
There are two consistent designs:
Status is implicit → then writing foo/status should be rejected with a clear message ("you don't need this, status comes with the parent claim").
Status is explicit → then foo/status claims should actually work in the authorizer.
The PR does neither: it accepts the claim syntactically but ignores it semantically. That's the worst combination, because the API lies to the user.
The suggested fix (cheap one):
Add a CEL validation so it's impossible to write:
rule: "!self.resource.endsWith('/status')"
message: "status is granted implicitly by the parent resource claim"
Now the user who writes cowboys/status gets an immediate, self-explanatory error at creation time instead of a silent no-op at request time.
There was a problem hiding this comment.
and we need e2e to for this
There was a problem hiding this comment.
The fix the AI suggested with not allowing the status subresource is what I mentioned in call earlier; but before we make a decision on that I think we should discuss it in the cc to see what people think.
Personally I would prefer having status implicit for this API version and not allow setting it and requiring it to be explicit in the next API version.
| // ScopedPermissionClaim embeds a PermissionClaim and adds a selector to | ||
| // scope down access to objects of the claimed resource. | ||
| // | ||
| // +kubebuilder:validation:XValidation:rule="!self.resource.contains('/') || (has(self.selector.matchAll) && self.selector.matchAll)",message="subresource claims must use selector.matchAll" |
There was a problem hiding this comment.
This is missleading a bit.
Maybe: "for subresource claims the selector is ignored; scope is inherited from the parent resource claim".
Summary
Implements subresources in VWs, specifically the
TokenRequest.Went with RBAC style in the claims for less API changes and because it feels closer. I tested a separate
Subresoureas part of the claims but that didn't look quite right.I've been hacking on this on and off for two weeks.
I think its secure now because to access a subresource a) the claim for the subresource needs to be present and b) the parent resource needs to be claimed as well, so the usual gates apply.
subresource claims cannot have their own labels to prevent mismatching labels between parent and subresource (I could see the value of being able to distinguish that a provider has access to resource set A but only to subset B for accessing the subresource - but that feels like the edge case of an edge case).
What Type of PR Is This?
/kind feature
Related Issue(s)
Fixes #
Release Notes